SPARK-3926 [CORE] Reopened: result of JavaRDD collectAsMap() is not serializable by srowen · Pull Request #3587 · apache/spark

srowen · 2014-12-03T20:03:00Z

My original 'fix' didn't fix at all. Now, there's a unit test to check whether it works. Of the two options to really fix it -- copy the Map to a java.util.HashMap, or copy and modify Scala's implementation in Wrappers.MapWrapper, I went with the latter.

SparkQA · 2014-12-03T20:10:31Z

Test build #24099 has started for PR 3587 at commit 12e4ecf.

This patch merges cleanly.

SparkQA · 2014-12-03T20:15:42Z

Test build #24099 has finished for PR 3587 at commit 12e4ecf.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-03T20:15:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24099/
Test FAILed.

SparkQA · 2014-12-03T20:35:23Z

Test build #24102 has started for PR 3587 at commit 7bb0e66.

This patch merges cleanly.

SparkQA · 2014-12-03T21:55:05Z

Test build #24102 has finished for PR 3587 at commit 7bb0e66.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-03T21:55:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24102/
Test PASSed.

zsxwing · 2014-12-04T02:52:36Z

core/src/main/scala/org/apache/spark/api/java/JavaUtils.scala

You can remove this() since you have already copying the codes.

It won't have a no-arg constructor in addition to its main one arg constructor otherwise right? This is the point of this change.

I ran your unit test after removing the no-arg constructor, and it worked fine.

Hm, so it does. Maybe I misunderstood the original error. It's complaining about the superclass (MapWrapper) not having a no-arg constructor? So copying the class works, since we no longer subclass MapWrapper, but the copy in SerializableMapWrapper need not define a no-arg constructor. OK, that line can be removed.

…ied code in LICENSE

SparkQA · 2014-12-05T15:35:13Z

Test build #24184 has started for PR 3587 at commit 8586bb9.

This patch merges cleanly.

zsxwing · 2014-12-05T15:41:13Z

LGTM

SparkQA · 2014-12-05T17:01:09Z

Test build #24184 has finished for PR 3587 at commit 8586bb9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-05T17:01:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24184/
Test PASSed.

JoshRosen · 2014-12-09T00:12:03Z

LGTM, too. Merging into master and tagging for branch-1.2 backport.

…erializable My original 'fix' didn't fix at all. Now, there's a unit test to check whether it works. Of the two options to really fix it -- copy the `Map` to a `java.util.HashMap`, or copy and modify Scala's implementation in `Wrappers.MapWrapper`, I went with the latter. Author: Sean Owen <sowen@cloudera.com> Closes #3587 from srowen/SPARK-3926 and squashes the following commits: 8586bb9 [Sean Owen] Remove unneeded no-arg constructor, and add additional note about copied code in LICENSE 7bb0e66 [Sean Owen] Make SerializableMapWrapper actually serialize, and add unit test (cherry picked from commit e829bfa) Signed-off-by: Josh Rosen <joshrosen@databricks.com>

JoshRosen · 2014-12-09T00:15:04Z

Merged to branch-1.1.

JoshRosen · 2014-12-17T20:14:01Z

I've merged this into branch-1.2.

…erializable My original 'fix' didn't fix at all. Now, there's a unit test to check whether it works. Of the two options to really fix it -- copy the `Map` to a `java.util.HashMap`, or copy and modify Scala's implementation in `Wrappers.MapWrapper`, I went with the latter. Author: Sean Owen <sowen@cloudera.com> Closes #3587 from srowen/SPARK-3926 and squashes the following commits: 8586bb9 [Sean Owen] Remove unneeded no-arg constructor, and add additional note about copied code in LICENSE 7bb0e66 [Sean Owen] Make SerializableMapWrapper actually serialize, and add unit test (cherry picked from commit e829bfa) Signed-off-by: Josh Rosen <joshrosen@databricks.com>

Make SerializableMapWrapper actually serialize, and add unit test

7bb0e66

srowen force-pushed the SPARK-3926 branch from 12e4ecf to 7bb0e66 Compare December 3, 2014 20:29

zsxwing reviewed Dec 4, 2014
View reviewed changes

Remove unneeded no-arg constructor, and add additional note about cop…

8586bb9

…ied code in LICENSE

asfgit closed this in e829bfa Dec 9, 2014

srowen deleted the SPARK-3926 branch December 9, 2014 17:15

JoshRosen mentioned this pull request Apr 2, 2015

SI-8911 scala.collection.convert.Wrappers$MapWrapper not serializable scala/scala#4167

Closed

Conversation

srowen commented Dec 3, 2014

Uh oh!

SparkQA commented Dec 3, 2014

Uh oh!

SparkQA commented Dec 3, 2014

Uh oh!

AmplabJenkins commented Dec 3, 2014

Uh oh!

SparkQA commented Dec 3, 2014

Uh oh!

SparkQA commented Dec 3, 2014

Uh oh!

AmplabJenkins commented Dec 3, 2014

Uh oh!

zsxwing Dec 4, 2014

Choose a reason for hiding this comment

Uh oh!

srowen Dec 4, 2014

Choose a reason for hiding this comment

Uh oh!

zsxwing Dec 4, 2014

Choose a reason for hiding this comment

Uh oh!

srowen Dec 5, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 5, 2014

Uh oh!

zsxwing commented Dec 5, 2014

Uh oh!

SparkQA commented Dec 5, 2014

Uh oh!

AmplabJenkins commented Dec 5, 2014

Uh oh!

JoshRosen commented Dec 9, 2014

Uh oh!

JoshRosen commented Dec 9, 2014

Uh oh!

JoshRosen commented Dec 17, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants